The Big Questions

• How do extract meaning from administrative data systems?
• Can we do data analysis fast enough to inform decisions and improve outcomes?
• Can we produce analyses that are approachable to policy makers the public so that they galvanize change?
• Can we do this under time and capacity constraints in a way that is rigorous, valid, and accurate?

Meet the Inference Tree

What is an inference tree?

• An inference tree isa way of dividing data into meaningful groups
• It is an algorithm that splits data by groups searching for the most meaningful and significant splits among grouping variables
• This is a way to identify meaningfully large groups with differing performance
• Useful for exploring policy opportunities

How can it be used?

• A little R code
• A little explanation
• A good dataset with good variables that are meaningful
• The good news is most states have all of this!

The R Code

• This looks remarkably like Stata code, but the plot drawing is much simpler in many cases

library(partykit)
mypar <- ctree_control(testtype = "Bonferroni", mincriterion = 0.99)
mytree <- ctree(mathSS ~ race + econ + ell + disab + sch_fay + dist_fay + attday + 
    readSS, data = subset(df, grade == 3))
plot(mytree)

Using Graphics To Tell a Story: Maps

• Data publicly available from Wisconsin Information Network for Successful Schools WINSS

Using Graphics to Tell a Story: Animations

• Not actual student data, simulated from Wisconsin DPI records for example purposes.

R Code

• This is not all the R code, animations are a bit more complicated
• This gives a feel for the structure
• The advantage is that they can be shared and reproduced like the SDP Toolkit graphics

# Draw subset
library(ggplot2)
for (i in 1:100) {
    mysub <- subset(studata, math_percentile == i)
    png(file = "frame", i, ".png", width = 1000, height = 750)
    qplot(grade, mathSS, data = mysub, geom = "lines", group = stuid, alpha = I(0.2)) + 
        prof_poly() + stat_summary(median, data = mysub, color = "gold", size = I(1.04)) + 
        geom_text(Labels)
    dev.off()
}

Demonstrate Distributions Linking Tests

• Simulated from ACT and WKCE data at Wisconsin Department of Public Instruction

This is different than research

• First, it is using the population data for what it should be used for
• Second, it’s free and easy to implement
• Third, it’s useful for questions policy makers might be eager to know–where should we focus our limited resources?
• So, how do we do it?

Ask Jen

• Every agency has someone who gets these requests, and that someone is overworked

  

The Problem?

• Jen doesn’t scale
• We don’t have enough Jen’s

Policy Relevant Research is the Answer

John Tukey said:

An approximate answer to the right problem is worth a good deal more than the exact answer to an approximate problem

• This is what policy relevant research is about

Fast

Focused

Approximate

Policy relevant research is everywhere

• COMPSTAT used by police departments around the world with an estimated 90% of large US police deparments implementing it http://www.policefoundation.org/pdf/growthofcompstat.pdf
• A/B testing at Khan Academy, Amazon, Netflix, and others
• Financial services, stock trading, etc.
• ROI on capital investments in municipal works
• Teaching is all about quick and focused research into what works through benchmark assessments and feedback

Getting Value from Data

• Over $500 millon spent on building Statewide Longitudinal Data Sysems at SEAs DQC
• Exponential increase in the amount of data collected
• Constrained resources, tightened budgets, lack of time
• Skills in program areas don’t align with data needs
• Researcher contracting does not scale, does not work on this timeline, and does not answer the question of the moment

How to do this?

• Everyone at an SEA has a ton of work and data analysis is often viewed as a luxury
• Additionally, work has to be done efficiently, tranparently, and reproducibly to be valid
• This is expensive in terms of time, money, and management resources
• Analyses that are done are done ad hoc and disconnected from a cohesive strategy
• They are poorly documented
• Done in disparate and proprietary toolsets
• Unable to be reproduced with updated data

• Knowledge is lost, not accumulated

• What are some solutions?

My Favorite

Wisconsin’s Solution

R

R is

• Free and open source
• Extensible
• Reproducible
• Scriptable
• Can be integrated into operational systems
• A gateway to learning statistics

But wait…

Isn’t R?

Confusing?

Buggy?

How does Wisconsin use R?

• R is used to script repeated tasks and free up agency staff time for more policy research
• AYP, or accountability, is calculated using an R script that is reproducible, transparent, and fast!
• Other reports are being autmoated as well, and can continue through staff turnover
• Data quality can be standardized with basic data checking modules run before every report
• This leaves more time to do policy relevant research, which R has some great tools for

How else is Wisconsin using R?

• Script common processes like AYP and other reports
• Fit growth models
• Visualize data for policy makers
• To build data indicators to use as discussions
• Analyze all sorts of ad hoc data from different sources, and combine datasets for quick policy questions

Some Caveats

• A pretty picture is not enough
• Good analysis is still not enough
• Need champions, policy makers, and decision makers who are willing to hear what the data say, what the data cannot say, and develop strategies to empirically investigate possible interventions/causes/policy relevant questions
• This is a tall order and one we haven’t quite squared in Wisconsin just yet

Thoughts

1. What are some ways inference trees could be used in your agency?
2. What types of analyses would be useful across district and state agencies?
3. Sharing work with common data is something we all have an interest in to understand our work. How can we make this happen?

Attribution and License

This work (Data Analysis in an SEA, by Jared E. Knowles), in service of the Wisconsin Department of Public Instruction, is free of known copyright restrictions.